The present invention relates generally to computer memory, and more specifically, to first-in-first-out (FIFO) queue-based command spreading.
Contemporary high performance computing main memory systems are generally composed of one or more memory devices, which are connected to one or more memory controllers and/or processors via one or more memory interface elements such as buffers, hubs, bus-to-bus converters, etc. The memory devices are generally located on a memory subsystem such as a memory card or memory module and are often connected via a pluggable interconnection system (e.g., one or more connectors) to a system board (e.g., a PC motherboard).
Overall computer system performance is affected by each of the key elements of the computer structure, including the performance/structure of the processor(s), any memory cache(s), the input/output (I/O) subsystem(s), the efficiency of the memory control function(s), the performance of the main memory devices(s) and any associated memory interface elements, and the type and structure of the memory interconnect interface(s).
Extensive research and development efforts are invested by the industry, on an ongoing basis, to create improved and/or innovative solutions to maximizing overall system performance and density by improving the memory system/subsystem design and/or structure. High-availability systems present further challenges as related to overall system reliability due to customer expectations that new computer systems will markedly surpass existing systems in regard to mean-time-between-failure (MTBF), in addition to offering additional functions, increased performance, increased storage, lower operating costs, etc. Other frequent customer requirements further exacerbate the memory system design challenges, and include such items as ease of upgrade and reduced system environmental impact (such as space, power and cooling). In addition, customers are requiring the ability to access an increasing number of higher density memory devices (e.g., DDR3 and DDR4 SDRAMs) at faster and faster access speeds.
In view of varying cost, capacity, and scalability requirements, a wide number of memory system options may need to be considered. Often a choice needs to be made between using an asynchronous boundary between a processor and memory buffer chips or designing a fully synchronous system. An asynchronous design allows the flexibility of running the processor at a fixed frequency, while memory buffer chips can be programmed to varying frequencies to match the desired memory components. For example, if cost is most important, a slower more widely available dual in-line memory module (DIMM) can be used. In contrast, if performance is paramount, then a leading edge technology DIMM can be used. This type of memory system architecture may work well in systems where each memory channel runs independently. However, this approach typically falls short in high-availability systems.
Redundant array of independent memory (RAIM) systems have been developed to improve performance and/or to increase the availability of storage systems. RAIM distributes data across several independent memory modules, where each memory module contains one or more memory devices. There are many different RAIM schemes that have been developed, each having different characteristics, and different pros and cons associated with them. Performance, availability, and utilization/efficiency (e.g., the percentage of the memory devices that actually hold customer data) are perhaps the most important. The tradeoffs associated with various schemes have to be carefully considered because improvements in one attribute can often result in reductions in another. Examples of RAIM systems may be found, for instance, in U.S. Patent Publication Number 2011/0320918 titled “RAIM System Using Decoding of Virtual ECC”, filed on Jun. 24, 2010, the contents of which are hereby incorporated by reference in its entirety, and in U.S. Patent Publication Number 2011/0320914 titled “Error Correction and Detection in a Redundant Memory System”, filed on Jun. 24, 2010, the contents of which are hereby incorporated by reference in its entirety.
High availability systems, such as RAIM systems, can include a number of clock domains in various subsystems. Efficient integration of subsystems including different clock domains presents a number of challenges to establish synchronization timing, detection of synchronization issues, and recovery of synchronization.
Various memory functions, such as command scheduling to the DRAM, power management, refresh and periodic calibration procedures may be moved from the memory control unit (MCU) to a buffer chip. Moving such functions to the buffer chip may impact utilization of the interface, or bus, between the MCU and the buffer chip. Utilization of the interface between the MCU and the buffer chip needs to be maintained at a high level to achieve good performance and memory throughput. A buffer chip may have multiple command schedulers, or DRAM ports, for parallel memory access. In multiple-port buffer chips with independent engines, it may be difficult to keep all ports as busy as possible and increase throughput of the buffer chip. Commands may be sent to the buffer chip from the MCU sequentially. However, this may cause one or more of the ports on the buffer chip to starve if, for example, a relatively large number of earlier, sequentially-received commands are sent to a first port on the buffer chip while a second port is idle, and a later-received command is then sent to the second port after the relatively large number of earlier-received commands.